Image processing method, image processing apparatus, imaging apparatus, and medium

ABSTRACT

An image processing method includes acquiring a captured image that has been generated by imaging through an imaging optical system inclined relative to an imaging plane, acquiring inclination information of the imaging optical system, acquiring aberration information of the imaging optical system, and performing image restoration processing on the captured image based on the aberration information. The image restoration processing includes a setting of a correction amount of the captured image according to the inclination information.

BACKGROUND Field of the Disclosure

The present disclosure relates to an image processing method forperforming image restoration processing of a captured image.

Description of the Related Art

Japanese Patent No. 5414752 discusses a method for performing imagerestoration processing by reconfiguring an optical transfer function ofan imaging optical system according to each position (position withrespect to a screen center or an optical axis of the imaging opticalsystem in one direction) in a captured image from coefficient datacorresponding to an imaging condition of the captured image. The opticaltransfer function at each of the positions in the captured image isrotated and developed around the optical axis of the imaging opticalsystem for use, which makes it possible to perform the image restorationprocessing rotationally symmetrical about the center of an imaging plane(screen center) or the optical axis while reducing an informationamount.

An imaging apparatus that uses an imaging optical system including atilt mechanism to perform tilt imaging has been known. In the tiltimaging, distortion due to perspective can be corrected by controllingan object plane to be focused thereon. The tilt imaging is imaging inwhich the imaging optical system is inclined (tilted) relative to theimaging plane, and it enables an entire object plane having a depth tobe in focus without increasing a depth of field, and enables a reductionof an in-focus range.

In the imaging optical system including the tilt mechanism, unlike atypical optical system rotationally symmetrical about the optical axis,image forming performance rotationally symmetrical about the center ofthe imaging plane or the optical axis is not necessarily obtainable. Inother words, rotationally asymmetrical eccentric aberration occurs in animage height direction with respect to the center of the imaging planeor the optical axis, and the image forming performance is deterioratedcompared with a reference state (not tilted).

When the image restoration processing rotationally symmetrical about thecenter of the imaging plane or the optical axis is performed on adeteriorated image, which is deteriorated due to the rotationallyasymmetrical aberration, obtained in the tilt imaging using the methoddiscussed in Japanese Patent No. 5414752, an adverse effect such asinsufficient correction, excessive correction, edge fall (undershoot),and ringing occurs because of a difference in aberration to becorrected. Japanese Patent No. 5414752, however, does not discuss ameasure against the adverse effect in a case where deterioration occursdue to the rotationally asymmetrical aberration in the tilt imaging. Inaddition, the optical transfer function corresponding to all positionsin the screen can be used in order to perform the image restorationprocessing for the rotationally asymmetrical aberration. However, theinformation amount is increased.

SUMMARY

The present disclosure is directed to an image processing method, animage processing apparatus, an imaging apparatus, and a medium capableof performing, on an image obtained by tilt imaging, image restorationprocessing suppressing an adverse effect while reducing an informationamount.

According to an aspect of the present invention, an image processingmethod includes acquiring a captured image that has been generated byimaging through an imaging optical system inclined relative to animaging plane, acquiring inclination information of the imaging opticalsystem, acquiring aberration information of the imaging optical system,and performing image restoration processing on the captured image basedon the aberration information. The image restoration processing includesa setting of a correction amount of the captured image according to theinclination information.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an image restoration filter accordingto each of exemplary embodiments.

FIG. 2 is a diagram illustrating the image restoration filter accordingto each of the exemplary embodiments.

FIG. 3 is a diagram illustrating a point spread function PSF accordingto each of the exemplary embodiments.

FIG. 4 is a diagram illustrating an amplitude component of an opticaltransfer function (OTF), i.e., a modulation transfer function (MTF) anda phase component of the OTF, i.e., a phase transfer function (PTF)according to each of the exemplary embodiments.

FIG. 5 is a diagram illustrating tilt imaging based on Scheimpflugprinciple according to an exemplary embodiment.

FIG. 6 is a flowchart illustrating an image processing method accordingto a first exemplary embodiment.

FIGS. 7A to 7E are diagrams illustrating a method for generating animage restoration filter according to the first exemplary embodiment.

FIG. 8 is a diagram illustrating an image processing system according toa second exemplary embodiment.

FIG. 9 is a block diagram illustrating an imaging apparatus according toa third exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention are described in detailbelow with reference to the drawings.

A captured image obtained by an imaging apparatus includes a blurcomponent due to influence of aberration such as spherical aberration,comatic aberration, curvature of field, and astigmatism of an imagingoptical system and thus the image quality thereof is deteriorated. Theblur component of the image due to such aberration indicates that anoptical flux that is emitted from one point of an object and that is tobe essentially collected again at one point of an imaging plane if thereis neither aberration nor influence of diffraction is spread. The blurcomponent is represented by a point spread function (PSF).

An optical transfer function (OTF) that is obtained by performingFourier transform on the PSF is frequency component information of theaberration and is represented in a complex number. An absolute value ofthe OTF, i.e., an amplitude component, is referred to as a modulationtransfer function (MTF), and a phase component is referred to as a phasetransfer function (PTF). The MTF (amplitude component) and the PTF(phase component) are respectively frequency characteristics of theamplitude component and the phase component of image qualitydeterioration due to the aberration, and are represented by thefollowing expression with the phase component as a phase angle.

PTF=a tan(Im(OTF)/Re(OTF))

In the expression, Re(OTF) and Im(OTF) respectively denote a real partand an imaginary part of the OTF. As described above, since the OTF ofthe imaging optical system causes deterioration of the MTF (amplitudecomponent) and the PTF (phase component) of the image, the deterioratedimage is in a state where points of the object are asymmetricallyblurred as with the comatic aberration. In addition, magnificationchromatic aberration occurs when an image forming position is shifteddue to a difference of image forming magnification between wavelengthsof light, and the magnification chromatic aberration is generated byacquiring image forming positions as, for example, red, green, and blue(RGB) color components according to a spectral characteristic of theimaging apparatus. Accordingly, the image forming positions are shiftedbetween RGB, and image spreading occurs in each of the color componentsdue to the difference of the wavelengths thereof, i.e., a phase shift.

As a method for correcting the deterioration of the MTF (amplitudecomponent) and the PTF (phase component), a correction method usingaberration information (OTF or PSF, and information relating thereto) ofthe imaging optical system is known. The method is referred to as imagerestoration or image recovery. Hereinafter, processing for correctingdeterioration of the captured image with use of the aberrationinformation of the imaging optical system is referred to as imagerestoration processing. As one of image restoration methods, a methodfor convolving the captured image with an image restoration filterhaving an inverse characteristic of the OTF is known, and a detailthereof is described below.

To effectively use the image restoration, it is necessary to obtainaccurate information of the OTF of the imaging optical system. The OTFof a typical imaging optical system largely varies with an image height(position of image). In addition, the OTF is two-dimensional dataconstituted of a complex number, which includes a real part and animaginary part. In a case where the image restoration processing isperformed on a color image including color components of three colorsRGB, the OTF at one image height includes the number of taps in avertical direction×the number of taps in a horizontal direction×2 (realpart and imaginary part)×3 (RGB). In the present disclosure, the numberof taps indicates a vertical size and a horizontal size of the OTF. Ifsuch taps are held for all imaging conditions such as an image height,an F-number (aperture value), a zoom position (focal length), and animage taking distance, a data amount becomes huge. Therefore, in anexemplary embodiment, a method and a configuration for reducing the dataamount (information amount) are described.

First, a definition of terms described in the present exemplaryembodiment and the image restoration processing (image processingmethod) are described. The image processing method described herein isappropriately used in each of exemplary embodiments described below.

<Captured Image>

A captured image is a digital image obtained by an imaging device thathas received light through an imaging optical system. The captured imageis deteriorated due to the OTF including aberration of the imagingoptical system, which includes a lens and various kinds of opticalfilters. The imaging optical system may also include a mirror(reflection surface) having curvature in addition to the lens.

Color components of the captured image include, for example, informationof RGB color components. In addition, a commonly-used color space suchas an LCH color space expressed in lightness, chroma, and hue and aYCbCr color space expressed in luminance and a color-difference signalmay also be selected and used. As the other color spaces, XYZ, Lab, Yuv,and JCh color spaces may be used. Further, color temperature may be usedas well.

To a captured image (input image) and an output image, an imagingcondition such as a focal length of a lens, an aperture value, and animage taking distance as well as various kinds of correction informationfor correcting the image may be added. In a case where the image istransferred from the imaging apparatus to another image processingapparatus to perform correction processing on the image, the imagingcondition and the information relating to correction are preferablyadded to the captured image as described above. As another method fortransferring the imaging condition and the information relating tocorrection, the imaging apparatus and the image processing apparatus maybe directly or indirectly connected to perform the transfer.

<Image Restoration Processing>

Subsequently, an outline of the image restoration processing isdescribed. When the captured image (deteriorated image) is denoted byg(x, y), an original image is denoted by f(x, y), and the PSF as aFourier pair of the OTF is denoted by h(x, y), Expression (1) isestablished.

g(x,y)=h(x,y)*f(x,y)  (1)

In this expression, * indicates convolution (convolution integration,product-sum), and (x, y) indicates a coordinate on the captured image.

Expression (1) is converted into an expression in an expression of afrequency plane through the Fourier transform to obtain Expression (2)represented by a product of each frequency.

G(u,v)=H(u,v)·F(u,v)  (2)

In this expression, H is the OTF obtained by performing the Fouriertransform on the PSF (h), G and F are functions obtained by performingthe Fourier transform on the deteriorated image g and the original imagef, respectively, and (u, v) is a coordinate on a two-dimensionalfrequency plane, i.e., a frequency.

To obtain the original image f from the captured deteriorated image g,both sides of Expression (2) are divided by the OTF H as represented byExpression (3).

G(u,v)/H(u,v)=F(u,v)  (3)

Then, inverse Fourier transform is performed on F(u, v), or G(u, v)/H(u,v), to return F(u, v) to a real plane, and the original image f(x, y) isobtained as a restored image.

When H⁻¹ subjected to the inverse Fourier transform is denoted by R,convolution processing is performed on the image on the real plane asrepresented by Expression (4), and the original image f(x, y) isobtained in a similar manner.

g(x,y)*R(x,y)=f(x,y)  (4)

In this expression, R(x, y) is referred to as an image restorationfilter. In a case where the image is a two-dimensional image, commonly,the image restoration filter R is also a two-dimensional filter havingtaps (cells) corresponding to pixels of the image. Further, restorationaccuracy is typically improved as the number of taps (number of cells)of the image restoration filter R is larger. Accordingly, the number ofachievable taps is set according to required image quality, imageprocessing capacity, a characteristic of aberration, etc. It isnecessary for the image restoration filter R to reflect at least thecharacteristic of aberration. Therefore, the image restoration filter Ris different from, for example, an existing edge enhancement filterincluding about three taps in each of a horizontal direction and avertical direction. Since the image restoration filter R is set based onthe OTF, the image restoration filter R can correct deterioration ofboth of the amplitude component and the phase component with highaccuracy.

Further, an actual image contains a noise component. Therefore, when theimage restoration filter R that is created from an inverse of the OTF asdescribed above is used, the noise component is significantly amplifiedalong with restoration of the deteriorated image. This is because theMTF (amplitude component) of the optical system is raised so as toreturn to 1 over the entire frequency in a state where amplitude of thenoise is added to the amplitude component of the image. The MTF(amplitude component), which is amplitude deterioration of the opticalsystem, is returned to 1; however, a power spectrum of the noise israised at the same time. As a result, the noise is amplified accordingto a degree of raising (restoration gain) of the MTF (amplitudecomponent).

Accordingly, in a case where the noise is contained, an image favorableas an image for appreciation is not obtained. This is represented byExpressions (5-1) and (5-2).

G(u,v)=H(u,v)·F(u,v)+N(u,v)  (5-1)

G(u,v)/H(u,v)=F(u,v)+N(u,v)/H(u,v)  (5-2)

where N is the noise component.

As for the image containing the noise component, a method forcontrolling a restoration degree according to an intensity ratio SNR ofan image signal and a noise signal may be used, for example, as with theWiener filter represented by Expression (6).

$\begin{matrix}{{M\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + {SNR}^{2}}}} & (6)\end{matrix}$

In this expression, M(u, v) is a frequency characteristic of the Wienerfilter, and |H(u, v)| is an absolute value (MTF) of the OTF. In thismethod, the restoration gain (restoration degree) is decreased as theMTF is smaller, and the restoration gain is increased as the MTF islarger, for each frequency. Typically, since the MTF of the imagingoptical system is high in a lower frequency and low in a higherfrequency, the restoration gain in the higher frequency of the image issubstantially reduced by this method.

Subsequently, the image restoration filter is described with referenceto FIGS. 1 and 2. The number of taps of the image restoration filter isdetermined according to an aberration characteristic of the imagingoptical system and required restoration accuracy. An image restorationfilter of FIG. 1 is a two-dimensional filter including 11×11 taps as anexample. In FIG. 1, a value (coefficient) of each of the taps is notillustrated, and a cross-section of the image restoration filter isillustrated in FIG. 2. Distribution of the values (coefficient values)of the taps of the image restoration filter has a function of returninga signal value (PSF) spatially spread due to the aberration ideally toan original one point.

Each of the taps of the image restoration filter is subjected toconvolution processing (convolution integration, product-sum)corresponding to each of the pixels of the image in a process of theimage restoration processing. In the convolution processing, to improvea signal value of a predetermined pixel, the predetermined pixel is madecoincident with the center of the image restoration filter. Further, aproduct of the signal value of the image and the coefficient value ofthe filter is calculated between corresponding pixels of the imagerestoration filter and the image, and a total sum of the productsreplaces a signal value of the center pixel.

Subsequently, a characteristic of the image restoration in a real spaceand a frequency space is described with reference to FIGS. 3 and 4. FIG.3 is an explanatory diagram of the PSF, where a left diagram of FIG. 3illustrates the PSF before the image restoration, and a right diagram ofFIG. 3 illustrates the PSF after the image restoration. FIG. 4 is anexplanatory diagram of the MTF (amplitude component) (left diagram ofFIG. 4) and the PTF (phase component) (right diagram of FIG. 4) of theOTF. A dashed line (A) in the left diagram of FIG. 4 indicates the MTF(amplitude component) before the image restoration, and an alternatelong and short dash line (B) indicates the MTF (amplitude component)after the image restoration. Further, a dashed line (A) in the rightdiagram of FIG. 4 indicates the PTF (phase component) before the imagerestoration, and an alternate long and short dash line (B) indicates thePTF (phase component) after the image restoration. As illustrated in theleft diagram of FIG. 3, the PSF before the image restoration has anasymmetrical spread, and due to the asymmetry, the PTF (phase component)has a non-linear value relative to the frequency. The image restorationprocessing performs correction such that the MTF (amplitude component)is amplified and the PTF (phase component) becomes zero. Therefore, thePSF after the image restoration has a symmetrical acute shape.

As described above, the image restoration filter can be obtained byperforming inverse Fourier transform on the function that is designedbased on the inverse function of the OTF of the imaging optical system.The image restoration filter used in the present exemplary embodimentcan be appropriately changed, and for example, the Wiener filter asdescribed above may be used. In a case of using the Wiener filter, theimage restoration filter of the real space with which the image is to beactually convolved can be obtained through inverse Fourier transform ofExpression (6). In addition, the OTF is varied depending on the imageheight (position of image) of the imaging optical system even in oneimaging state. Therefore, the image restoration filter to be used ischanged depending on the image height.

Next, tilt imaging according to the present exemplary embodiment isdescribed with reference to FIG. 5. FIG. 5 is an explanatory diagram ofthe tilt imaging based on the Scheimpflug principle. In the tiltimaging, the imaging optical system is inclined (tilted) relative to theimaging plane, which causes a principle plane LPP of the imaging opticalsystem to be inclined relative to an imaging plane IP. In other words,the captured image according to the present exemplary embodiment is animage acquired in a state where the principle plane LPP of the imagingoptical system is inclined relative to the imaging plane IP of theimaging device.

At this time, an object plane OP on which the imaging optical system isfocused, the principle plane LPP of the imaging optical system, and theimaging plane IP intersect with one another on a straight line P. Insuch tilt imaging, the object plane OP to be focused thereon is inclinedrelative to the imaging plane IP, which makes it possible to control anin-focus range irrespective of an aperture value (F-number) of theimaging optical system. For example, it enables the entire object planeOP having a depth to be in focus without reducing an aperture value toincrease a depth of field. In contrast, extremely narrowing the in-focusrange allows for diorama-like image representation.

The tilt imaging can be achieved by providing a tilt mechanism on theimaging optical system. Further, the imaging optical system may includea revolving mechanism that makes a tilt direction variable. In the tiltimaging, eccentric aberration occurs because the object plane OP, theprinciple plane LPP of the imaging optical system, and the imaging planeIP are not parallel to one another. The eccentric aberration refers toan eccentric coma (eccentric comatic aberration), eccentric distortion,a color shift caused by eccentricity, and the like. Such eccentricaberration in the tilt imaging occurs as aberration rotationallyasymmetrical about the center of the imaging plane IP or an optical axisOA of the imaging optical system.

Next, the image processing method according to a first exemplaryembodiment of the present invention is described with reference to FIG.6. FIG. 6 is a flowchart of the image processing method (imageprocessing program) according to the present exemplary embodiment. Theimage processing method according to the present exemplary embodiment isexecuted by a computer, which includes a central processing unit (CPU)and the like, as the image processing apparatus according to the imageprocessing program as a computer program. This also applies to otherexemplary embodiments described below.

First, in step S11, the image processing apparatus acquires the capturedimage that has been generated by the imaging apparatus through imaging.The captured image may be acquired from the imaging apparatus throughwired or wireless communication between the imaging apparatus and theimage processing apparatus or through a storage medium such as asemiconductor memory and an optical disk. Next, in step S12, the imageprocessing apparatus acquires an imaging condition (imaging conditioninformation) at a time when the imaging apparatus generates the capturedimage through the imaging. As described above, the imaging conditionincludes identification information (camera identification (ID)) of theimaging apparatus in addition to the focal length, the aperture value(F-number), and the image taking distance of the imaging optical system.In addition, in the imaging apparatus in which the imaging opticalsystem is interchangeable, the imaging condition may also includeidentification information (lens ID) of the imaging optical system(interchangeable lens). The imaging condition information may beacquired as attendant information to the captured image as describedabove or may be acquired through wired or wireless communication or astorage medium.

Subsequently, in step S13, the image processing apparatus acquiresinformation indicating a state of the tilt imaging (i.e., informationindicating a tilt state of the imaging optical system) at the time whenthe imaging apparatus generates the captured image through the imaging.In acquiring the information indicating the state of the tilt imaging,the image processing apparatus first determines whether the imagingapparatus has performed the tilt imaging, i.e., whether the imagingoptical system has been tilted relative to the imaging plane. In a casewhere it is determined as the tilt imaging, the image processingapparatus acquires information relating to a tilt direction (inclinationdirection) and a tilt angle (inclination angle) relative to the imagingplane of the imaging optical system in the tilt imaging state. The tiltdirection and the tilt angle are respectively illustrated by an arrowand θ in FIG. 5. Hereinafter, the information indicating the state ofthe tilt imaging is referred to as tilt information. The tiltinformation may be acquired as attendant information to the capturedimage or may be acquired through wired or wireless communication or astorage medium as with the above-described imaging conditioninformation.

Subsequently, in step S14, the image processing apparatus acquiresaberration information suitable for the imaging condition. In thepresent exemplary embodiment, the aberration information is the OTF. Theimage processing apparatus selects and acquires, from the plurality ofOTFs held in advance, a suitable OTF according to the imaging condition.In addition, in a case where the imaging condition such as the aperturevalue, the image taking distance, and the focal length of the zoom lensis a specific imaging condition, an OTF corresponding to the specificimaging condition may be generated through interpolation processing fromthe OTFs of the other imaging conditions held in advance. In this case,it is possible to reduce the data amount of the OTFs to be held. As theinterpolation processing, for example, bilinear interpolation (linearinterpolation) and bicubic interpolation are used; however, theinterpolation processing is not limited thereto.

In the present exemplary embodiment, the image processing apparatusacquires the OTF as the aberration information; however, the aberrationinformation is not limited thereto. The image processing apparatus mayacquire the aberration information such as the PSF in place of the OTF.Moreover, in the present exemplary embodiment, the image processingapparatus may acquire approximated coefficient data through fitting theaberration information to a predetermined function, and may reconfigurethe OTF and the PSF based on the coefficient data. For example, the OTFmay be fitted with use of the Legendre polynomial. In addition, the OTFmay be fitted with use of another function such as the Chebyshevpolynomial.

In step S14, the image processing apparatus (OTF acquisition unit)generates the plurality of OTFs in one direction that passes through thescreen center (center of the captured image) or the optical axis OA ofthe imaging optical system and is perpendicular to the optical axis OA.The imaging optical system may include an imaging device, an optical lowpass filter, etc.

Subsequently, in step S15, the image processing apparatus (OTFdevelopment unit) rotates the OTFs around the screen center (center ofthe captured image) or the optical axis OA of the imaging optical systemto develop the OTFs. More specifically, the image processing apparatusinterpolates the OTFs corresponding to a pixel array to discretelydispose the OTFs to a plurality of positions in the captured image.

Subsequently, in step S16, the image processing apparatus (imagerestoration filter generation unit) converts the OTFs into the imagerestoration filters (filters). In other words, the image processingapparatus generates the image restoration filters with use of thedeveloped OTFs. The image restoration filter is generated by creating arestoration filter characteristic in the frequency space based on theOTF and by converting the restoration filter characteristic in thefrequency space into a filter (image restoration filter) in the realspace through inverse Fourier transform.

Steps S15 and S16 are described in detail with reference to FIGS. 7A to7E. FIGS. 7A to 7E are explanatory diagrams of the method for generatingthe image restoration filter. As illustrated in FIG. 7A, the OTFs aredisposed in one direction (vertical direction) that passes through thescreen center (center of the captured image) or the optical axis OA ofthe imaging optical system and is perpendicular to the optical axis OAwithin a region of a circumscribed circle of the image (imaging region).

In the present exemplary embodiment, in step S14 of FIG. 6, the OTFs aredeveloped on a straight line as illustrated in FIG. 7A; however, thedevelopment is not limited thereto. For example, in the captured imageplane, straight lines that pass through the center of the captured imageor the optical axis OA of the imaging optical system and are orthogonalto each other are referred to as a first straight line (y in FIG. 7A)and a second straight line (x in FIG. 7A). At this time, it issufficient for at least two of the OTFs acquired in step S14 tocorrespond to positions (image height) on the first straight line. Inother words, the OTFs do not have to be linearly disposed in onedirection as long as the OTFs are disposed at a plurality of positions(plurality of positions within the captured image) arranged in apredetermined direction at different distances from the screen center orthe optical axis OA of the imaging optical system. In a case where apixel including the center of the captured image or the optical axis OAof the imaging optical system does not exist, i.e., in a case where thecenter of the captured image or the optical axis OA of the imagingoptical system is located between pixels, it is sufficient for the OTFsacquired in step S14 to correspond to the positions (image height) ofthe pixels interposing the first straight line.

Further, in a case where the OTFs are arranged in one direction, thedirection is not limited to the vertical direction, and the OTFs mayalso be arranged in another direction such as a horizontal direction.The OTFs are preferably linearly arranged in any of the verticaldirection and the horizontal direction because the image processingaccording to the present exemplary embodiment can be more easilyperformed.

Subsequently, the OTFs are rotated, and the interpolation processing(various kinds of processing according to pixel arrangement afterrotation) is performed as necessary to rearrange the OTFs as illustratedin FIG. 7B. The interpolation processing includes interpolationprocessing in a radiation direction and interpolation processingassociated with the rotation, and allows for rearrangement of the OTFsat arbitrary positions. Next, for example, the frequency characteristicof the image restoration filter is calculated as represented byExpression (6), and inverse Fourier transform is performed on the OTF ateach position to perform conversion into the image restoration filter inthe real space as illustrated in FIG. 7C.

In other words, in the captured image, the straight lines that passthrough the center of the captured image or the optical axis OA of theimaging optical system and are orthogonal to each other are referred toas the first straight line y in FIG. 7A and the second straight line xin FIG. 7A. A region point-symmetrical to a first region 73 of thecaptured image in FIG. 7C about the center of the captured image or theoptical axis OA of the imaging optical system is referred to as a secondregion 71 in FIG. 7C. Further, a region line-symmetrical to the firstregion 73 about the first straight line y is referred to as a thirdregion 72 in FIG. 7C, and a region line-symmetrical to the first region73 about the second straight line x is referred to as a fourth region 74in FIG. 7C. At this time, the OTFs of the second region 71, the thirdregion 72, and the fourth region 74 are generated with use of the OTFsof the first region 73. As a result, an amount of Fourier transformprocessing is reduced to approximately ¼ of that of finally-rearrangedpositions. In addition, rearranging the OTFs of FIG. 7B and the imagerestoration filters of FIG. 7C through rotation and interpolationprocessing as illustrated in FIG. 7E and developing the OTFs and theimage restoration filters with use of symmetry as illustrated in FIG. 7Dmake it possible to further reduce the amount of Fourier transformprocessing. The arrangement (arrangement density of the restorationfilters) illustrated in FIGS. 7A to 7E are examples, and an arrangementdistance may be arbitrarily set according to variation of the OTFs ofthe imaging optical system.

In the present exemplary embodiment, the OTFs arranged in one directionthat passes through the screen center or the optical axis OA of theimaging optical system are rotated and developed on an assumption thatthe OTFs are rotationally symmetrical about the center of the imagingplane (screen center) or the optical axis OA of the imaging opticalsystem. This makes it possible to perform the image restorationprocessing with a small amount of data. The image processing apparatusholds in advance, as the OTFs, the OTFs in one direction that passesthrough the screen center (center of the captured image) or the opticalaxis OA of the imaging optical system in a state where the imagingoptical system is not tilted.

The eccentric aberration in the tilt imaging, however, occurs asaberration rotationally asymmetrical about the center of the imagingplane (screen center) or the optical axis OA of the imaging opticalsystem. If the image restoration processing on an assumption of therotationally symmetrical aberration according to the present exemplaryembodiment is performed on the captured image deteriorated by therotationally asymmetrical aberration in the tilt imaging, an adverseeffect such as insufficient correction, excessive correction, edge fall(undershoot), and ringing occurs because the aberration to be correctedis different.

Accordingly, the image processing apparatus according to the presentexemplary embodiment determines a restoration gain of the imagerestoration filter based on the tilt information. More specifically, theeccentric aberration is increased and a difference with the rotationallysymmetrical aberration in a no-tilt state is increased as the tilt angleθ is increased. Therefore, the restoration gain is set smaller as thetilt angle θ is larger. In other words, the image processing apparatussets a correction amount (restoration gain) to a first correction amount(first restoration gain) in a case where the tilt angle θ is a firstinclination angle θ1. In addition, in a case where the tilt angle θ is asecond inclination angle θ2 that is larger than the first inclinationangle θ1, the image processing apparatus sets the correction amount(restoration gain) to a second correction amount (second restorationgain) that is smaller than the first correction amount. For example, theimage processing apparatus sets a restoration gain A=2 cos θ for thetilt angle θ. The restoration gain A may be a maximum value, an averagevalue, or a value at a specific frequency of the image restorationfilter in the frequency space. The present exemplary embodiment is notlimited thereto.

Moreover, in a case where the tilt angle θ in the imaging is equal to orlarger than a set threshold, the restoration gain may be set smallerthan the restoration gain corresponding to the tilt angle θ that issmaller than the set threshold. In other words, in a case where the tiltangle θ is smaller than the predetermined threshold, the imageprocessing apparatus sets the correction amount (restoration gain) to athird correction amount (third restoration gain). In addition, in a casewhere the tilt angle θ is equal to or larger than the predeterminedthreshold, the image processing apparatus sets the correction amount(restoration gain) to a fourth correction amount (fourth restorationgain) smaller than the third correction amount.

The image processing apparatus according to the present exemplaryembodiment preferably determines the restoration gain based on anaperture value (F-number). When the aperture value is increased, arotationally symmetrical diffraction phenomenon becomes more dominantthan the rotationally asymmetrical aberration in the tilt imaging.Therefore, the restoration gain is made smaller as the aperture value issmaller. In other words, in a case where the aperture value is a firstaperture value, the image processing apparatus sets the correctionamount (restoration gain) to a fifth correction amount (fifthrestoration gain). Further, in a case where the aperture value is asecond aperture value smaller than the first aperture value, the imageprocessing apparatus sets the correction amount (restoration gain) to asixth correction amount (sixth restoration gain) smaller than the fifthcorrection amount.

Alternatively, in a case where the aperture value in the imaging islarger than a minimum aperture value of the imaging optical system, therestoration gain may be set larger than the restoration gain at theminimum aperture value. In other words, in a case where the aperturevalue is equal to the minimum aperture value of the imaging opticalsystem, the image processing apparatus sets the correction amount(restoration gain) to a seventh correction amount (seventh restorationgain). Further, in a case where the aperture value is larger than theminimum aperture value of the imaging optical system, the imageprocessing apparatus sets the correction amount (restoration gain) to aneighth correction amount (eighth restoration gain) larger than theseventh correction amount.

Furthermore, in a case where the aperture value in the imaging is equalto or smaller than a set threshold, the restoration gain may be setsmaller than the restoration gain corresponding to the aperture valuelarger than the set threshold. In other words, in a case where theaperture value is larger than the predetermined threshold, the imageprocessing apparatus sets the correction amount (restoration gain) to aninth correction amount (ninth restoration gain). In addition, in a casewhere the aperture value is equal to or smaller than the predeterminedthreshold, the image processing apparatus sets the correction amount(restoration gain) to a tenth correction amount (tenth restoration gain)smaller than the ninth correction amount.

Subsequently, in step S17 of FIG. 6, the image processing apparatus(image restoration unit) executes the image restoration processing ofthe captured image with use of the image restoration filters generatedin step S16. In other words, the image processing apparatus convolvesthe captured image with the image restoration filters to perform theimage restoration processing of the captured image. Then, in step S18,the image processing apparatus acquires a restored image based on aresult of the image restoration processing in step S17.

In the present exemplary embodiment, the restoration gain of each of theimage restoration filters has been determined based on the tiltinformation in step S16; however, the restoration gain in the imagerestoration processing may also be determined based on the tiltinformation. The restoration gain in the image restoration processingmay be adjusted when a difference between the images before and afterthe image restoration processing is added to the image before the imagerestoration processing to change a ratio in acquisition of the restoredimage. In other words, as the tilt angle θ is larger, the ratio of thedifference between the images before and after the image restorationprocessing to be added to the image before the image restorationprocessing is made small to make the restoration gain in the imagerestoration processing smaller. Further, the restoration gain in theimage restoration processing may also be determined based on theaperture value in the above-described manner.

In the convolution of the image restoration filters, pixels at positionsother than positions where the image restoration filters are disposedillustrated in FIG. 7D may be interpolated and generated with use of theplurality of filters disposed in the vicinity thereof. At this time, theimage restoration filters include a first image restoration filter at afirst position of the captured image and a second image restorationfilter at a second position of the captured image. The first imagerestoration filter is generated with use of the developed opticaltransfer function. The second image restoration filter is generatedthrough interpolation using the first image restoration filter. Suchinterpolation processing enables to change the image restoration filter,for example, for each pixel.

In the present exemplary embodiment, the case where the processing basedon the inverse function of the OTF is performed as the image restorationprocessing to correct deterioration of the image has been described. Inaddition, unsharp mask processing using the aberration information mayalso be applied to the present exemplary embodiment . . . .

In the unsharp mask processing, a difference between an original imageand an unsharp image that is obtained by applying an unsharp mask to theoriginal image is added to or subtracted from the original image togenerate a sharpened image. At this time, the PSF of the imaging opticalsystem is used as the unsharp mask to acquire the image in whichdeterioration due to the aberration of the imaging optical system in theimaging has been corrected.

When the captured image is denoted by g(x, y) and a correction componentis denoted by u(x, y), the corrected image f(x, y) is represented byExpression (7).

f(x,y)=g(x,y)+m×u(x,y)  (7)

In Expression (7), varying a value m makes it possible to adjust therestoration degree (restoration gain) of the correction component u(x,y), i.e., the correction amount, of the captured image g(x, y). Thevalue m may be varied according to the image height (position of theimage) of the imaging optical system or may be a fixed value.

In addition, the correction component u(x, y) is represented byExpression (8).

u(x,y)=g(x,y)−g(x,y)*PSF(x,y)  (8)

Further, the correction component u(x, y) is represented by Expression(9) by deforming a right side of Expression (8).

u(x,y)=g(x,y)*(δ(x,y)−PSF(x,y))  (9)

In Expression (9), δ is a delta function (ideal point image). The deltafunction used here is data in which the number of taps is equal to thatof the PSF(x, y), a tap value at the center is one, and all the othertap values are zero.

The corrected image f(x, y) is represented by Expression (10) fromExpressions (7) to (9).

f(x,y)=g(x,y)*[δ(x,y)+m×(δ(x,y)−PSF(x,y))]  (10)

In other words, the captured image g(x, y) is convolved with a part insquare brackets in Expression (10) as the filter (image restorationfilter) to perform the unsharp mask processing.

The PSF is varied according to the image height (position of the image)of the imaging optical system even in one imaging state. Therefore, thefilter of the unsharp mask processing is also changed and used accordingto the image height. The restoration gain of the unsharp mask processingis also determined based on the tilt information and the aperture valueas described above. Further, various improved resolution processing(resolution enhancement processing) such as super-resolution processingusing the aberration information is similarly applicable.

Next, an image processing system including the image processingapparatus that performs the above-described image processing method isdescribed with reference to FIG. 8. FIG. 8 is an explanatory diagram ofan image processing system 100 according to a second exemplaryembodiment. The image processing system 100 includes an aberrationinformation calculation apparatus 101, a camera 110 (imaging apparatus),and an image processing apparatus 120.

The aberration information calculation apparatus 101 performs processingto calculate the OTF from a design value or a measured value of theimaging optical system according to the imaging condition of thecaptured image. The OTF to be calculated here is the OTF in a statewhere the imaging optical system is not tilted. The camera 110 includesan imaging device 111 and an imaging lens 112. The camera 110 adds, tothe image captured by the imaging lens 112, a lens ID and imagingcondition information (such as an aperture value, a zoom position, andan image taking distance) of the imaging lens 112, a Nyquist frequencyof the imaging device 111, and tilt information, and outputs theresultant image. For example, in a case where the tilt imaging isperformed (the imaging optical system includes a tilt mechanism), thetilt angle θ may be detected by an angle detector including an encoderand may be reflected in the tilt information. At this time, the tiltangle θ may be detected as a relative angle of the imaging opticalsystem to the imaging plane. Further, a tilt direction may be detected.

The image processing apparatus 120 includes an image restorationinformation holding unit 121, an aberration information acquisition unit122, and a filter processing unit (image acquisition unit and imagerestoration processing unit) 123. The image processing apparatus 120holds information output from the aberration information calculationapparatus 101 and the camera 110, and uses the information to correctthe deteriorated image captured by the imaging lens 112 (i.e., performsimage restoration processing of the captured image).

The image restoration information holding unit 121 stores information ofthe OTF calculated by the aberration information calculation apparatus101, the number of taps, the lens ID, the imaging condition, and theNyquist frequency of the imaging device for each of combinations of theimaging lens 112 and the imaging device 111. As described above, theimage restoration information holding unit 121 is a storage unit storingthe OTF corresponding to the imaging condition of the captured image.Further, the image restoration information holding unit 121 storesrestoration gain information based on the tilt information. Moreover,the image restoration information holding unit 121 may also storerestoration gain information based on the aperture value. For example,the image restoration information holding unit 121 creates and stores,in advance, a table of a restoration gain value corresponding to thetilt angle θ and the aperture value. Alternatively, the imagerestoration information holding unit 121 may store information in whichthe restoration gain corresponding to the tilt angle θ and the aperturevalue is converted into a function.

The aberration information acquisition unit 122 acquires, from thecamera 110, the Nyquist frequency information of the imaging device 111and the captured image and the lens ID and the imaging conditioninformation of the imaging lens 112. The aberration informationacquisition unit 122 searches through the OTFs saved in the imagerestoration information holding unit 121 based on the lens ID and theimaging condition of the camera 110 used by a photographer in theimaging. Then, the aberration information acquisition unit 122 acquiresa corresponding OTF (OTF suitable for the lens ID and the imagingcondition in the imaging). The aberration information acquisition unit122 acquires the OTF used by the filter processing unit 123 in thespatial frequency domain up to the Nyquist frequency of the imagingdevice 111 of the camera 110. In other words, the aberration informationacquisition unit 122 uses the acquired OTF to acquire the OTF of theimaging optical system (imaging lens 112) corresponding to the positionof the captured image. As described above, the aberration informationacquisition unit 122 serves as an OTF acquisition unit that acquires theOTF of the imaging optical system corresponding to the position of thecaptured image. Further, the aberration information acquisition unit 122serves as an OTF development unit that develops the OTF by rotating theOTF around the center of the captured image or the optical axis OA ofthe imaging optical system.

The filter processing unit 123 acquires the captured image from thecamera 110. In other words, the filter processing unit 123 correspondsto an image acquisition unit. The filter processing unit 123 uses theOTF acquired by the aberration information acquisition unit 122 tocreate the image restoration filter for correction of deterioration ofthe captured image, thereby correcting deterioration of the image. Inother words, the filter processing unit 123 serves as a filtergeneration unit that generates the image restoration filter with use ofthe developed OTF. Further, the filter processing unit 123 correspondsto an image restoration processing unit that performs the imagerestoration processing on the captured image with use of the imagerestoration filter. The filter processing unit 123 acquires the tiltinformation and the restoration gain information corresponding to thetilt angle θ and the aperture value stored in the image restorationinformation holding unit 121, and determines the restoration gain basedon the tilt information. Alternatively, the filter processing unit 123may determine the restoration gain based on the aperture value. In acase where the table of the restoration gain value corresponding to thetilt angle θ and the aperture value is used as the restoration gaininformation, the filter processing unit 123 may determine therestoration gain corresponding to the tilt information and the aperturevalue through interpolation of the table.

When the image restoration information holding unit 121 holds the OTFcalculated in advance by the aberration information calculationapparatus 101, it is not necessary to provide the aberration informationcalculation apparatus 101 to a user (photographer). Further, the usermay use information necessary for the image restoration processing suchas coefficient data by downloading the information through a network orvarious kinds of storage media.

Next, an imaging apparatus according to a third exemplary embodiment ofthe present invention is described with reference to FIG. 9. FIG. 9 is ablock diagram of an imaging apparatus 200 according to the presentexemplary embodiment. An image processing program that performs imagerestoration processing (image restoration method similar to that in thefirst exemplary embodiment) of the captured image is installed in theimaging apparatus 200, and the image restoration processing is executedby an image processing unit 204 (image processing apparatus) inside theimaging apparatus 200.

The imaging apparatus 200 includes an imaging optical system 201 (lens)and an imaging apparatus main body (camera main body). The imagingoptical system 201 includes a tilt mechanism, a diaphragm 201 a, and afocus lens 201 b, and is integrally configured with the imagingapparatus main body (camera main body). However, the present exemplaryembodiment is not limited thereto and is also applicable to an imagingapparatus in which the imaging optical system 201 is interchangeablymounted on the imaging apparatus main body.

An imaging device 202 photoelectrically converts an object image(optical image or image forming light) formed through the imagingoptical system 201 to generate the captured image. In other words, theobject image is photoelectrically converted by the imaging device 202into an analog signal (electric signal). Then, the analog signal isconverted into a digital signal by an analog/digital (A/D) converter203, and the digital signal is input to the image processing unit 204.

The image processing unit 204 (image processing apparatus) performspredetermined processing on the digital signal and performs theabove-described image restoration processing. The image processing unit204 includes an image acquisition unit 204 a, an aberration informationacquisition unit 204 b, and an image restoration processing unit 204 c.The image acquisition unit 204 a and the image restoration processingunit 204 c have the function of the filter processing unit 123 accordingto the second exemplary embodiment. The aberration informationacquisition unit 204 b has the function of the aberration informationacquisition unit 122 according to the second exemplary embodiment.

First, the image processing unit 204 acquires the imaging conditioninformation of the imaging apparatus from a state detection unit 207.The imaging condition information is information relating to theaperture value (F-number), the image taking distance, the focal lengthof the zoom lens, and the like. The state detection unit 207 can acquirethe imaging condition information directly from a system controller 210;however, the configuration is not limited thereto. For example, theimaging condition information relating to the imaging optical system 201may be acquired from an imaging optical system control unit 206.Further, the state detection unit 207 acquires the tilt information inthe imaging. The process flow (image processing method) of the imagerestoration processing according to the present exemplary embodiment issimilar to that in the first exemplary embodiment described withreference to FIG. 6, and a description thereof is therefore omitted.

The OTF is held in a storage unit 208. The output image processed by theimage processing unit 204 is saved in an image recording medium 209 in apredetermined format. A display unit 205 displays an image obtained byperforming predetermined display processing on the image subjected tothe image restoration processing according to the present exemplaryembodiment. The image to be displayed is not limited thereto, and animage subjected to simplification processing for high-speed display maybe displayed on the display unit 205.

A series of control in the present exemplary embodiment is performed bythe system controller 210, and the imaging optical system 201 ismechanically driven by the imaging optical system control unit 206 inresponse to an instruction from the system controller 210. The imagingoptical system control unit 206 controls an aperture diameter of thediaphragm 201 a as an imaging state setting of the aperture value(F-number). Further, the imaging optical system control unit 206controls a position of the focus lens 201 b by an unillustratedautomatic focus (AF) mechanism or an unillustrated manual focusmechanism to adjust a focus according to an object distance. Functionssuch as the aperture diameter control of the diaphragm 201 a and themanual focus may not be executed depending on a specification of theimaging apparatus 200.

The imaging optical system 201 may include an optical device such as alow pass filter and an infrared cut filter; however, if a device such asthe low pass filter that influences a characteristic of the OTF is used,some consideration is necessary at a time of creating the imagerestoration filter in some cases. The infrared cut filter alsoinfluences each of the PSFs of RGB channels that are integral values ofthe PSFs of spectral wavelengths, in particular, the PSF of the Rchannel. Therefore, some consideration is necessary at the time ofcreating the image restoration filter in some cases. In this case, asdescribed in the first exemplary embodiment, the rotationallyasymmetrical transfer function is added after the OTFs are rearranged.

In the present exemplary embodiment, the OTF stored in the storage unit208 of the imaging apparatus is used. As a modification, the imagingapparatus may acquire the OTF stored in a storage medium such as amemory card.

As described above, in each of the exemplary embodiments, the imageprocessing apparatus 120 (image processing unit 204) includes the filterprocessing unit 123 (image acquisition unit 204 a and image restorationprocessing unit 204 c) and the aberration information acquisition unit122 (204 b). The filter processing unit 123 (image acquisition unit 204a) acquires the captured image generated by the imaging through theimaging optical system that is inclined (tilted) relative to the imagingplane IP. The aberration information acquisition units 122 and 204 bacquire the aberration information (restoration gain) of the imagingoptical system. The filter processing unit 123 (image restorationprocessing unit 204 c) acquires inclination information (tiltinformation) of the imaging optical system, and performs the imagerestoration processing on the captured image based on the aberrationinformation. Further, the filter processing unit 123 (image restorationprocessing unit 204 c) sets the correction amount (the restoration gainor the restoration amount) of the captured image according to theinclination information.

The present invention may be achieved by processing in which a programachieving one or more functions of the above-described exemplaryembodiments is supplied to a system or an apparatus through a network ora storage medium, and one or more processors in a computer of the systemor the apparatus read and execute the program. Further, the presentinvention may also be achieved by a circuit (e.g., application specificintegrated circuit (ASIC)) achieving one or more functions.

According to the exemplary embodiments, it is possible to provide theimage processing method, the image processing apparatus, the imagingapparatus, and the medium capable of performing, on the image obtainedby the tilt imaging, the image restoration processing suppressing anadverse effect while reducing the information amount.

Other Embodiments

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™,a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-156780, filed Aug. 15, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing method, comprising: acquiringa captured image generated by imaging through an imaging optical systeminclined relative to an imaging plane; acquiring inclination informationof the imaging optical system; acquiring aberration information of theimaging optical system; and performing image restoration processing onthe captured image based on the aberration information, wherein theimage restoration processing includes a setting of a correction amountof the captured image according to the inclination information.
 2. Theimage processing method according to claim 1, wherein the correctionamount is a restoration gain in the image restoration processing.
 3. Theimage processing method according to claim 1, further comprisingacquiring an imaging condition of the captured image, wherein theaberration information is aberration information corresponding to theimaging condition.
 4. The image processing method according to claim 1,wherein the aberration information is aberration information of theimaging optical system in a state where the imaging optical system isnot inclined relative to the imaging plane.
 5. The image processingmethod according to claim 1, wherein the aberration information isacquired by rotating and developing aberration information around acenter of the captured image or an optical axis of the imaging opticalsystem, the aberration information relating to one direction that passesthrough the center of the captured image or the optical axis of theimaging optical system and is perpendicular to the optical axis.
 6. Theimage processing method according to claim 1, further comprisingcreating a filter based on the aberration information, wherein the imagerestoration processing includes image restoration processing using thefilter.
 7. The image processing method according to claim 1, wherein theinclination information includes information relating to an inclinationangle of a principle plane of the imaging optical system to the imagingplane, wherein in the image restoration processing, the correctionamount is set to a first correction amount in a case where theinclination angle is a first inclination angle, and the correctionamount is set to a second correction amount smaller than the firstcorrection amount in a case where the inclination angle is a secondinclination angle larger than the first inclination angle.
 8. The imageprocessing method according to claim 1, wherein the inclinationinformation includes information relating to an inclination angle of aprinciple plane of the imaging optical system relative to the imagingplane, and wherein, in the image restoration processing, the correctionamount is set to a third correction amount in a case where theinclination angle is smaller than a predetermined threshold, and thecorrection amount is set to a fourth correction amount smaller than thethird correction amount in a case where the inclination angle is equalto or larger than the predetermined threshold.
 9. The image processingmethod according to claim 1, wherein the image restoration processingincludes determining the correction amount according to the inclinationinformation and an aperture value in the imaging.
 10. The imageprocessing method according to claim 9, wherein, in the imagerestoration processing, the correction amount is set to a fifthcorrection amount in a case where the aperture value is a first aperturevalue, and the correction amount is set to a sixth correction amountsmaller than the fifth correction amount in a case where the aperturevalue is a second aperture value smaller than the first aperture value.11. The image processing method according to claim 9, wherein, in theimage restoration processing, the correction amount is set to a seventhcorrection amount in a case where the aperture value is equal to aminimum aperture value of the imaging optical system, and the correctionamount is set to an eighth correction amount larger than the seventhcorrection amount in a case where the aperture value is larger than theminimum aperture value of the imaging optical system.
 12. The imageprocessing method according to claim 9, wherein, in the imagerestoration processing, the correction amount is set to a ninthcorrection amount in a case where the aperture value is larger than apredetermined threshold, and the correction amount is set to a tenthcorrection amount smaller than the ninth correction amount in a casewhere the aperture value is equal to or smaller than the predeterminedthreshold.
 13. The image processing method according to claim 1, whereinthe aberration information is an optical transfer function of theimaging optical system.
 14. The image processing method according toclaim 1, wherein the aberration information is a point spread functionof the imaging optical system, and wherein the image restorationprocessing includes unsharp mask processing on the captured image withuse of the point spread function.
 15. An image processing apparatus,comprising: an image acquisition unit configured to acquire a capturedimage generated by imaging through an imaging optical system inclinedrelative to an imaging plane; an aberration information acquisition unitconfigured to acquire aberration information of the imaging opticalsystem; and an image restoration processing unit configured to performimage restoration processing on the captured image based on theaberration information and to acquire inclination information of theimaging optical system, wherein the image restoration processing unitsets a correction amount of the captured image according to theinclination information.
 16. An imaging apparatus, comprising: animaging device configured to photoelectrically convert an optical imageformed through an imaging optical system; and an image processing unit,wherein the image processing unit includes: an image acquisition unitconfigured to acquire a captured image generated by imaging through theimaging optical system inclined relative to an imaging plane of theimaging device; an aberration information acquisition unit configured toacquire aberration information of the imaging optical system; and animage restoration processing unit configured to perform imagerestoration processing on the captured image based on the aberrationinformation and to acquire inclination information of the imagingoptical system, wherein the image restoration processing unit sets acorrection amount of the captured image according to the inclinationinformation.
 17. A medium storing a program for causing a computer toexecute the image processing method according to claim 1.